Why your web server needs a data mart

The other day, I had a fun discussion with a colleague. We talked about data marts. Just in case you are not familiar with the term, let me take a moment to describe the concept.

A data mart is basically a small database that contains a sub-set of your company’s data. It is a copy.  On the surface, it might sound like a waste of time. After all, maintaining two sets of data, can be challenging and it is an opportunity for mistakes.  So why bother?


Most IT systems (programs) store their data in some kind of database. Some systems are (primarily) meant for gathering data and some are meant for displaying data. Big companies tend to have very large databases. Getting useful data out of them can be slow. Inserting data into a big DB can slow-down other people who are trying to read data. On a web site, you don’t want “slow”. There are of course, ways around this.

Managing/Dealing-with Large Data

When you think about data, it is best to think of your data like your money. Keeping it is good. The more (useful data) you gather, the more power it can provide to you. You always need some of it at-hand. Once you gather a bunch, you probably want to start putting some away (like a 401k). Your goal will be to accumulate it steadily. The expectation is that it will be more valuable someday. Having a good plan is important, so you can get a good value out of it someday.

Here is something that you probably don’t want to do: carry all of your money around with you all of the time. Why? Because you don’t want someone to steal it, it is probably big and bulky, and you don’t really need it right now. Just put it in the bank and you can get to it when you need it.

Data is like this too. On your web server, it is pretty rare for someone to need all of the data that your company has been accumulating. You usually only need current data, or a few summaries, and maybe sometimes, you need a big chunk, but not all of the time.

Solution: Data Mart

A data mart is basically, like a wallet full of data or maybe a debit card for data or something like that. It is a smaller database that only contains the data that your web server is typically going to use on a day, for one system. You don’t have your debit card connected to your 401 k, right? You also don’t need your web server connected to all of your data. Your data is also probably pretty bulky and slow. Maybe bad people would like to do evil stuff with it, so you should protect it and only carry-around the data that you actually need.

To summarize, a data mart for your web server, benefits you by:

  • Isolating traffic – web server demand is isolated from your main DB. So all web traffic doesn’t affect your main DB, and traffic to your main DB doesn’t slow-down your web server. This protects you against a DDOS attack.
  • Smaller data = faster – Certainly, it is much faster to query a smaller amount of data. This protects you against normal, healthy traffic and yields quick responses.
  • Less data = less exposure – If your web server ever becomes compromised (hacked), the culprits are likely to get everything that is on the server, and might get everything connected to it. If you plan your systems for this possibility, you will see that this defensive posture (of having less data) minimizes the damage which could occur from a data breach.

Bottom line: keep different (isolated) systems for your internal users, and external users. It takes a little more thinking, planning, and equipment, but it is much better than walking around with a big sack of money.

Posted in Architecture, Database, Optimization | Tagged , , , | Leave a comment

Value of a senior developer – part 3

In part 1, I talked about some unusual interview questions that were posed to me: 1) What does a company get from a senior developer for all of that extra cost? 2) You have a lot of experience with leadership, but we don’t need that. Can you just be a really good developer and not try to lead?

I suppose in parts 1 & 2, I mostly went-into: why those are not good questions, but didn’t exactly answer them or explain. So, here is a quick list of things that I would expect a senior developer to know, which go above the skill-set of a mid-level developer.

  • Start a project (correctly)
  • Finish a project (correctly)
  • Provide reliable estimates
  • Operate in a predictable/reliable manner
  • Assess other developers
  • Assess the skills of a business analyst, tester or project manager (objectively)
  • Be self-aware and know his own strengths/weaknesses and play to each of them
  • Advise management about options (technologies, dev practices, security, logic, paperwork)
  • Inherently prepare contingency plans
  • Anticipate risks and other problems in a project
  • Know when to go cowboy (in emergencies) and when to be patient
  • Know other technical people’s jobs: DBA, server admin, security admin, software architect
  • Show good judgment about which work should be done by the senior developer, and which work should be delegated
  • The buck stops with him, with-regard-to software development & problem solving
  • Know when something is over his head and when to get help

I could add a lot of other skills to this list, but I think you get the picture. A senior developer is not just someone with a lot of experience, or gray hair, but rather, he is someone who thoroughly knows software development projects and can entirely handle things himself. He is the leader and he is reliable. He knows what to do. He also can fill missing/empty roles if/when necessary.

If any of the skills on that list don’t seem like a big-deal to you, then you should speak with a senior developer about what goes wrong, when a person cannot perform those tasks. I could write two anecdotal stories about each of them, to prove how important they are, and why they are harder than they sound. The more skilled/experienced your senior developer, the higher-quality you will get out of each of those. Your projects should purr like a Cadillac.

Btw, if you have projects that don’t go-well, and you are not sure why, then you might want to use this list as an evaluation “check list”. Use it to evaluate your projects and see how many of them are contributing to your problems.

Posted in Career, Professionalism | Tagged , , , | Leave a comment

Emerging from technical bankruptcy

The process for emerging from technical bankruptcy is actually similar to financial bankruptcy.  If you decide to pay-down the technical debt, instead of starting over, it means you need to change your current practices.  You have been living beyond your means and you have been focusing your resources in the wrong place.

To actually pay-down your tech debt, you absolutely must cut-back on other projects and assign a higher priority to the process of paying-down-debt. If I’m making it sound easy, let me correct that notion.  This is going to require a significant effort.

On the other hand, if you think you will do better by starting over, think again.  Before you try it, you really need to take a deep look at yourself and your processes, so you don’t arrive at the same point, over-and-0ver again.

It is a long road back and it begins with things like “living within your means”, being realistic about your ability to produce and maintain software.  Committing to improving your processes and paying attention to quality.

It is a lot of work and a lot of commitment, but you can do it. Keep your head above water.  Live within your technical means.  Don’t accumulate technical debt.  Stick with these rules and some day, you won’t have to spend all of your resources, paying interest. Remember: Accumulating debt and never paying it down, will lead to one place: bankruptcy.  Don’t go there.


Posted in Methodology | Tagged , , | Leave a comment

Management will prefer more tech debt (in so-many words)

Disclaimer: This is not about my current project or company. It is about all projects and companies, everywhere.

If you are familiar with “technical debt”, I’m sure you would acknowledge: tech debt is bad, and if you allow it, you might never be able to get rid of it. So it makes you wonder why a team would accumulate any of it in the first place?

Believe it or not, technical debt is a tempting proposition for management.  To understand why, let’s first look at the ways to prevent tech debt.  Preventing tech debt, is really just a matter of finding & resolving tech debt very early in the process.  The sooner you resolve it, the lower your chance of accumulating it.  This is accomplished by reviewing code often, and enforcing standards:

  • Developer self-reviews and refactoring
  • Peer reviews
  • Formal code reviews
  • Architecture and system reviews
  • Quarterly and annual upgrades to technology
  • An overall attitude of constant improvement

If you are a manager, you probably read that list and thought, “Wow, that is a list of increasingly lofty goals.” If you were a VP, you probably read the same list and thought, “We can live without all that stuff. Cut it from scope”.

The reason, is because each one takes time. The purpose of the review, is to find things to improve, and by “improve”, I really mean “change”.  Changes mean re-testing, which usually means more bugs and more re-testing.  It adds risk, and management has trouble seeing the risk + reward.  However, they can easily see the risk. After-all, the if the system is already running (as-is), then there must not be a problem, right?  This is why management feels a strong incentive to ignore the reward for resolving tech-debt. Therefore, you can see how [accumulating technical debt], would seem like a tempting proposition. You could spend that money(time) elsewhere.

Reward for the risk

First, to be clear: all of the reviewing in the world, won’t make a bit of difference if you cannot/will-not commit-to resolving the problems that are found during a review.  It is like seeing a house fire, saying “someone should put-out that fire” and then walking away, without doing anything to help.

Every bit of tech debt is a metaphorical fire hazard.  If you lived in a house where some of the electrical work was not up-to-code, you might be willing to live with it.  However, there are building codes for a reason.  Each of those violations is a drop-in-a-bucket, or a straw on a camel’s back.  It may be difficult to discern where you turn the corner from “a few building-code violations” to “this is a death-trap”.  You might not realize it until something bad happens.  When it does, then everyone is suddenly an expert and “knew this would happen”.

Okay, so it is tempting, but dangerous.  Now what?

In reality, your IT management needs to define these policies. Hopefully, they realize this and assign a reasonable tech-debt ceiling (like a credit limit) and “monthly payments” (code-reviews, and monthly tech-debt resolution budget).  If not, then send them my way and have them read a few articles.  Then encourage them to goog/bing it for a 2nd opinion.

Credit companies understand debt and debt management.  They require monthly payments, and have credit limits.  Since monetary debt has many similarities to tech-debt, you may want to consider similar practices.

Posted in IT Psychology, Lessons Learned, planning | Tagged , | Leave a comment

Terms & conditions screen in RBA

Disclaimer: As always, I’m not supposed to talk about this SDK, but the accompanying docs do not cover the material very well, and bing/goog returns zero info.  I had to spend several hours figuring this out (and almost gave-up, twice).  Hopefully, this might save someone a few hours, or prevent someone from giving-up.   So, if this seems a little vague to you, it is intentional. Please just ignore it and carry-on.

We needed a means to display a bunch of information to a user/customer.  The default screens, and confirmation screen only allow 7 lines.  That would not be enough for my current needs.  Digging through the SDK docs, I found something called the TC.K3Z screen.  It sounded great.  I could send a “M25” message and show reams of text.  One little problem: I can’t send the text as a parameter to the M25 command.  It had to be part of a TC#.XML file.

I dug around until I found an example: the existing TC1.XML file.  The content was really simple. I already know how to upload a file, using the P62 params, and the ProcessMessage() call.  The only problem is that, all of my examples have a M97 command afterwards. Without M97, it doesn’t seem to pick up the new file/settings. I don’t really want to wait for a M97 to complete (40 sec).  I was fortunate to discover that you DO NOT need to do a M97 after uploading a TC#.XML file.  It grabs a fresh copy every time.  That is good news.

Last hurdle: message formatting.  Since the TC file was XML, clearly it would choke on greater-than and less-than symbols.  Those can be escaped using HTML escape sequences (&lt; for <, and &gt; for >).  However, line breaks didn’t work. at. all.  I tried <br> and I even tried &lt;br&gt;, like I found in the SECURPROMPT.XML file.  Nope.  It didn’t work and it hid any content like that.  Since the display was line-breaking at 70 chars per line, I was afraid that I might have to resort to line splitting and padding (add spaces until every line was 70 chars long). When I tried that, the display was adding an extra space at the beginning of every line except the first.  Weird!  When I removed all CRLFs, the extra space went away. Fortunately I found a better way for all of it.

For some reason I tried the literal of “\n” and discovered that would actually cause a line-break to be displayed on the display. So I did a find/replace of CRLF with the string \n.


This line will display\nmulti-line messages which are &lt; 70 chars long.

Displays as:

This line will display
multi-line messages which are < 70 chars long

I’m not sure why they would design it like that.  It is like the folks who designed it, sort-of mashed some HTML and C escape sequences, and just figured everyone would find that to be intuitive.  *sigh*

Anyway, that solved it for me.  I hope you found this useful and it saved you some time.

Posted in Lessons Learned, Programming | Tagged | Leave a comment

If someone asks you to use [StupidTechnologyDujour]

I had a fun conversation with a peer today. He asked me what I would say if a boss or peer wanted to use [Stupid-Technology-Dujour] (replace with technology of your choice). I know he was asking a rhetorical question, but I felt compelled to answer it honestly, “it depends”. He quickly corrected me, “the only right answer, is NO” (rather firmly). I didn’t get a chance to explain my answer and he didn’t ask, but I think the discussion revealed a little about each of us.

Throughout my career, I’ve sought-out career and technology advice. There are always an abundance of new technologies popping-up all over the IT community. Many of them are not practical for my current projects, but sound exciting.  I know that I can’t possibly try all of them, in my free time.  So I am grateful that I can ask some of my friends and peers about their experiences with those technologies.

I get to hear, from people whom I trust, which technologies are working better/worse, what kind of problems (and solutions) are associated to each, and the overall value that can be harvested from these technologies. It is so much easier to learn this information from trustworthy people, and not just fanboys or hype-men. The trick is: discerning which of my friends & colleagues have advice which is accurate and valuable. Much of this comes-down-to the personality and character of the person who is offering the advice.

Advice from Paper Tigers

I once worked with a guy who had a PHD, and taught (for several years) at a major university overseas. He had some very impressive credentials and he was glad to wave them around, and demand respect from others. I still remember the time he offered me this advice, “SharePoint is terrible. If anyone recommends using it, you must resist them in every way possible”. I had to laugh inside, because I know SharePoint really well. It is a great product, if you use it for the correct purpose. It can even work well for the wrong purpose, as long as you have some ninja-skills for SharePoint. So, his advice revealed a little something about him: He was the kind of guy who jumped to conclusions and made definitive statements about technologies, which he barely understood (or wrongly thought he understood). Basically, he was wrong. It wasn’t the last time either. He offered plenty of bad advice to me (and everyone in my department).


Making a good decision

Let me return to my first thought. If somebody asks me to use [Stupid-technology-dujour], I don’t just automatically blurt-out an emphatic “yes sir!” I take a moment to think about a few things before I decide if a technology is even worth a hoot.

There are 3 factors which are pretty influential to me, in choosing a technology:

  1. Maturity: How easy is it to find support for this technology (how big is the dev community and what is the overall opinion of it)? I’ve lived the nightmare of using a rare bleeding-edge technology, in-hopes that it will catch-on and I could claim the honor of being an early adopter. It was scary and exhausting. Those gambles have never paid-off for me. I’m more conservative now, but not overly-so (imo).
  2. Popularity: Do I personally know somebody who is already using this technology, and has experience? What kind of stories does this person have to say about it? Are there any people working with me (here) who are using it? How hard is it to find a rent-a-ninja for this technology?
  3. Adoption/learning-curve: Are the people here receptive to learning this new technology? Can they do it?

Maybe also: “Does this technology have a good reputation, or is it still bleeding-edge stuff? How much risk does it bring?”

Clearly, you will not do well with any technology if you already have some reservations/fear about it. Certainly, if you just have a closed mind, you will not be able to overcome an obstacle. It reminds me of a Henry Ford quote, “Whether you say you can, or you cannot, you are correct”.  So be realistic about that factor.

For a resourceful and intelligent person, nearly anything can be accomplished with enough determination and time. Therein lies my answer, “it depends”. You just need to determine whether you have enough intelligence, resources, determination and time. The rest, is a decision to invest those here, or elsewhere.

Posted in Lessons Learned, Programming | Tagged , , , , , | Leave a comment

Metal detector, for needles-in-a-haystack

A few weeks back, I had some good advice for someone on stack overflow https://stackoverflow.com/questions/49817985/htmlagilitypack-exception-has-been-thrown-when-used-in-deployed-ssis-package?noredirect=1#comment86732223_49817985

The guy was stuck with a bug and couldn’t figure it out. Sometimes, you can’t get any help on Stack Overflow because people are like “can we see your code” or “can you describe the root cause better”. However, if you can’t seem to get to the root cause, or “the problem is in there somewhere, but I don’t know where”, then you probably won’t get much help on S/O.  You’ve got to feel a little bad for someone in that condition, but it is hard to help without enough info.

My advice to this guy was: “add lots of debugging and tracing, to try and capture an error message or get some idea of where it was failing and why”. He didn’t seem satisfied at first, but eventually he gave it a try. Two days later, his error handling caught the exact problem and told him where to look, etc. At that point, the solution was obvious/easy. I got a nice little “thank you” out of it, but no S/O points (and no additional street-cred). That actually happens a lot on S/O, because I often find folks like this guy on S/O and give them advice about how to solve it themselves.

Moral of this story

I have to tell you this: (Write it down if you have trouble remembering) … error handling, logging and tracing are the most important things that you can add into any program to make it more supportable. In contrast, if you do not have error handling, logging and tracing, you better pray that you never have to support that app and never have a big outage in production. I don’t want to sound overly dramatic, but I’m telling you, [error handling, logging and tracing] is nearly a religion for me. I’m not kidding. If you even try to remove error handlers or trace lines from my program, I might rough-you-up. Okay, I won’t clean your clock, but I will lose my friendly-demeanor for a moment and you will probably get an earful.

If you are new to programming or don’t yet have an appreciation for what I’m saying, please learn more on-your-own, or if you are on one of my projects, please get with me, so I can put the fear of “production outages” into you.

Ninja school

I honestly wish that universities had a mandatory course for CSE majors called “Maintainability”.  It would be an entire semester, where you take your old programs (from previous courses), and put error handlers around everything, and add trace points, and your trace (etc.) could be toggled on/off by a config.  You would know what to put in a config and you would know when to store telemetry in a DB or a file, or somewhere else.  You would make little utility apps to help you diagnose problems and read/aggregate your logs & traces.

For the final project, your projects would be subjected to the worst IT s#!+storm outage-fest that an app could face, and you would be graded on your ability to determine if the problem was 1) missing/revoked permissions (yours, or a service account), 2) bad network slowness, 3) TCP failure or router hiccup, 4) unexpected server/client upgrade which killed a dependent library (DLL), 5) full HDD on a server, or a few other spicy plot twisters.  Woo, I’m getting goose-bumps just thinking about it.

Error handling, logging and tracing are pre-requisite skills, if you ever intend to call yourself a ninja, or alpha geek, or the-go-to-guy.  If you don’t already eat, sleep and breathe it, then you should get out from under that rock and start learning it.  Start ‘leeting up, now.

Posted in Errors, Lessons Learned, Programming | Tagged , , , | Leave a comment